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Introduction 

It  is  axiomatic  that  evaluation  plays  a  major  role  in  all  phases  of  the  systematic  approach 
to  improving  human  performance.  Implementing  performance  interventions  without 
following  up  through  evaluation  means  that  positive  results,  though  hoped  for,  cannot  be 
directly  verified.  In  such  a  case,  an  organization  may  train,  create  job  aids,  buy  new 
equipment  or  institute  any  number  of  other  interventions  without  the  means  to  determine 
whether  these  "investments"  actually  improved  performance  and  provided  a  reasonable 
"return." 

Because  training  is  unquestionably  among  its  more  resource-intensive  and  pervasive 

interventions,  the  Coast  Guard  recently  analyzed  the  evaluation  of  training  to  establish 
policies  and  procedures  to  ensure  that  its  training  efforts  are  effective.  This  paper 
describes  the  Coast  Guard's  newly  implemented  training  evaluation  policy  and 
associated  responsibilities  and  infrastructure. 

The  Case  of  Action 

The  Coast  Guard  sponsors  over  700  resident  and  non-resident  training  courses  offered  by 
various  training  providers.  The  term  "training  provider"  in  this  paper  refers  to  any 
source  that  delivers  training  to  Coast  Guard  members  (e.g.,  Coast  Guard  Training 
Centers,  contracted  training  vendors,  and  other  government  agencies  and  military 
services).  Historically,  evaluation  methods  have  varied  among  training  providers  with 
many  providers  conducting  only  the  most  rudimentary  student  critiques.  Moreover,  data 
received  from  evaluations  were  not  sufficiently  communicated  to  Program  Managers  or 
Training  Consultants,  the  organization-level  reviewers  who  make  key  decisions 
regarding  training  resource  allocation.  Without  a  steady  stream  of  evaluation  data,  these 
reviewers  have  little  or  no  information  to  identify  trends  and  ensure  training  is  "hitting 
the  mark'  in  terms  of  improving  or  enabling  performance. 

Seeking  to  remedy  this  situation,  the  Coast  Guard  adopted  the  widely  accepted  four-level 
evaluation  model  first  introduced  by  Dr.  Donald  Kirkpatrick  some  four  decades  ago.  A 
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brief  explanation  of  these  levels  is  included  in  Figure  1.  The  levels  are  ordered  in  the 
sequence  in  which  they  are  assessed;  each  level  provides  valuable  feedback  on  distinct 
aspects  of  the  linkage  between  training  and  performance.  The  first  two  levels,  referred  to 
as  "Internal  evaluations,"  are  internal  to  the  training  providers  and  directly  affect  the 
design,  development,  delivery  and  administration  of  training.  The  second  two  levels, 
known  as  "external  evaluations,"  are  external  to  the  training  providers  and  focus  on  the 
impact  of  training  on  job  and  organizational  performance. 


The  Four  Levels  of  Training  Evaluation 

(The  Kirkpatrick  Model) 

Level 

Definition 

Advantages 

Disadvantages 

Level  1- 

"Reaction" 

Measures  trainees* 
opinions  about  the 
course.  This  Is  the 
most  common  way 
to  evaluate  student 
reaction  to  the 
course  and  provides 
a  measure  of 
immediate  customer 
satisfaction  with 
content,  delivery, 
and  environmental 
factors.  Often 
referred  to  as  "Smile 
Sheets." 

Low  cost  and  easy 
to  administer. 
Provides  insights  to 
participant's 
personal  feelings 
about  the  course. 
Provides  quick 
feedback  on 
successes  and 
failures  to  the 
training  provider. 

Only  reflects  a 
quick  reading  of  the 
participant  while 
they  are  still  in  the 
class.  Results 
should  not  be  used 
as  a  solid  basis  for 
changing  the 
educational  content 
or  strategy 

Level  2- 

"Leaming" 

Measures  how  well 
participants  have 
mastered  the  course 
objectives.  Can 
include  tests  of 
performance 
immediately  before 
and  after  the  course. 

Compared  to  level 

1 ,  this  provides 
more  compelling 
evidence  of  whether 
the  training  program 
works. 

Requires  more  time 
and  money  than 
level  1 .  Also 
requires  greater 
insight  to  the 
evaluation  process 
to  develop  valid 
measures  of 
learning. 
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Level  3- 

"Behavior" 

Assesses  practical 
value  of  training. 
Measures  how  well 
the  knowledge, 
skills,  and/or  values 
from  the  course  are 
used  in  the  job. 
Typically  measured 
3-6  months  after  the 
course. 

Provides  stronger 
evidence  that  the 
investment  in 
training  yields  the 
desired  return.  If 
designed  properly, 
can  also  identify 
barriers  and 
obstacles  to 
improved 
performance. 

Requires 

significantly  more 
investment  of  time 
and  money. 

Requires  in-depth 
insight  into 
performance 
interventions  and 
root  causes  of 
performance 
deficiencies. 

Level  4- 

"Results" 

Measures 

performance 

improvement, 

quality 

improvements,  and 
cost  savings  to  the 
organization. 
Measures  the  return 
on  investment  of  the 
training  course. 

Provides  strong 
evidence  that 
training  program  has 
impact  on 
organization. 
Addresses  whether 
the  performance  is 
important  to  the 
organization's 
bottom  line  (e.g., 
production,  safety, 
sales). 

Substantial  levels  of 
investment  and 
expertise  are 
required  to  develop 
level  4.  Often  hard 
to  decide  whether  or 
not  this  level  is 
required.  Linkage 
from  training  to 
org'l  results  is  hard 
to  establish. 

Figure  1:  The  Four  Levels  of  Training  Evaluation 


Policy  Analysis 

Substantial  research  and  analysis  were  conducted  prior  to  developing  any  polices  or 
making  any  sweeping  changes  to  the  existing  informal  training  evaluation  performance 
consultants  and  a  full-time  graduate  student  spent  over  a  year  examining  related 
literature  and  best  practices.  Below  are  observations  and  conclusions  based  on  the 
research,  benchmarking,  and  analysis  that  helped  shape  a  new  Coast  Guard-specific 
training  evaluation  policy  and  infrastructure. 

1 .  Level-  1  and  -2  evaluations  are  clearly  the  responsibility  of  the  training  providers, 
who  are  most  knowledgeable  of  the  training  environment  and  the  trainees'  reactions. 

The  training  providers  have  sole  control  over  how  they  administer  and  deliver  the 
course  objectives  and  whether  or  not  the  participants  acquire  the  knowledge,  skills,  or 
values  during  training.  Coast  Guard  Training  Centers  are  staffed  with  civilian  and 
military  professionals  well  versed  in  the  Instructional  Systems  Design  (ISD)  process  and 
are  very  capable  of  handling  levels  land  2.  General  guidelines  are  all  that  is  needed  to 
ensure  uniformity  of  practice. 

2.  Level-3  evaluations  require  a  standardized  process  to  ensure  the  results  are 
communicated  consistently  and  regularly  to  the  appropriate  decision-makers.  Under  the 
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existing  system,  many  training  providers  were  not  even  conducting  level-3  evaluations. 
Where  these  evaluations  were  conducted,  there  were  problems  including: 

0  There  were  no  criteria  for  using  and  analyzing  evaluation  data. 

0  Questions  and  evaluation  forms  were  non-standard  and  often  did  not  accurately 
reflect  the  course  content. 

0  The  evaluations  were  not  built  into  the  training  process  and  therefore  were 
inconsistent.  Identifying  negative  trends  and  making  decisions  on  remedial  performance 
interventions  were  virtually  impossible. 

0  Program  Managers  and  Training  Consultants-those  who  make  decisions  affecting 
whether  courses  are  maintained,  deleted,  or  modified-did  not  receive  evaluation  data. 

3.  Some  level-4  evaluations  are  in  place.  There  is  a  significant  body  of  literature  that 
indicates  level  4  is  not  required  for  all  types  of  training.  Moreover,  it  is  difficult  to 
isolate  the  contribution  of  any  one  determinant  such  as  training  in  achieving 
organizational  results.  Finally,  measurement  of  some  training  courses  and  their 
competency  objectives  would  either  be  impossible  or  would  yield  invalid  and 
inconclusive  results.  For  example,  even  if  Coast  Guard  EMTs  never  used  CPR,  the 
course  would  still  be  deemed  necessary.  Level  4  requires  more  research  before  making 
changes.  As  with  levels  I  and  2,  general  guidelines  will  assist  Program  Managers  in 
continuing  to  assess  the  ultimate  worth  of  training. 


A  Focus  on  Level  3 


In  view  of  the  research  results,  it  was  clear  that  special  emphasis  needed  to  be  placed  on 
level-3  evaluation  efforts,  which  were  not  meeting  the  needs  of  the  Coast  Guard's 
training  system.  Correcting  this  shortfall  required  a  completely  different  approach.  A 
formal  mechanism  was  established  that  revolved  around  a  standard  survey  instrument 
that  would  assess  the  on-the-job  impact  of  training.  The  entire  process  to  collect  course 
data,  distribute  post-graduation  surveys,  collect  responses,  and  report  results  to 
appropriate  consumers  would  be  centralized  to  provide  for  standardization,  efficiency, 
and  consistency. 

Level-3  Evaluation  Administration 

Due  to  its  resident  expertise  in  surveys  and  capability  for  mass  mail  distribution,  the 
Coast  Guard  Institute,  located  in  Oklahoma  City,  was  selected  as  the  administrative  hub 
for  level-3  evaluations.  To  this  end,  the  Coast  Guard  Institute  is  now  responsible  for 
maintaining  a  database  of  course  completion  dates  and  graduates  to  trigger  mailing  of 
the  right  surveys  to  the  right  people  at  the  right  times.  In  this  database,  the  Institute  will 
maintain  a  library  of  instruments  that  have  been  tailored  to  individual  courses.  Six 
months  after  the  completion  of  a  course,  the  Institute  will  send  a  survey  to  each  trainee 
and  the  trainee's  supervisor.  Concerns  about  sampling  validity  are  minimal  since  the 
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entire  target  population  will  be  surveyed. 

Once  the  survey  is  returned  to  the  institute,  the  data  will  be  captured  in  a  report  to 
Program  Managers  and  Training  Consultants  providing  frequency  of  responses. 

Program  Managers  and  Training  Consultants  can  request  additional  reports  to  correlate 
specific  variables  (e.g.,  unit  to  unit,  supervisors  to  trainees,  class  to  class  comparisons). 
These  reports  will  be  used  to  make  decisions  on  whether  in-depth  analysis  is  required  to 
determine  causes  for  inadequate  transfer  of  training  to  the  job.  Undesirable  survey 
results  would  require  such  an  analysis  since  a  variety  of  barriers  (possibly  having 
nothing  to  do  with  the  way  the  training  was  conducted)  might  prevent  training  from 
having  its  intended  impact. 

To  support  this  policy  administration,  two  new  data-entry  positions  will  be  added  to  the 
Institute's  staff  Moreover,  an  evaluation  expert  will  be  added  to  the  Coast  Guard 
Headquarters  Office  of  Training  and  Performance  Consulting  to  oversee  the  overall 
process 

The  Level-3  Survey  Instrument 

The  team  of  survey  developers  recognized  the  unfeasibility  of  using  the  instrument  to 
identify  the  myriad  possible  barriers  to  human  performance.  Instead,  the  level-3 
instrument  will  serve  as  an  indicator  or  "dashboard  gauge"  to  help  identify  areas 
requiring  a  more  detailed  analysis  of  the  transfer  of  training  to  job  behavior. 

Initially,  it  was  thought  that  the  course  objectives  themselves  could  be  directly  inserted 
into  the  instrument;  however,  this  resulted  in  redundancies  and  poorly  worded  items 
unsuitable  for  a  survey.  As  a  result,  training  providers  were  charged  with  development 
of  "performance  indicators"  that  reflect  the  ultimate  job  performance  objectives  of  each 
course.  The  survey  framework  into  which  these  performance  indicators  fit  is  depicted  in 
Figure  2.  Designated  respondents  receive  a  cover  sheet  with  the  survey  explaining  the 
instrument  and  providing  instructions  for  completion. 

Conclusion 

The  successful  implementation  of  the  four-level  evaluation  system  required  a  new  policy 
facilitated  by  an  innovative  infrastructure.  The  policy  outlines  responsibilities  at  all 
levels  in  the  evaluation  process  providing  informal  guidelines  for  levels  1,  2  and  4  and 
formal  mandates  for  level  3.  The  Coast  Guard  Institute  provides  primary  infrastructure 
support.  The  new  evaluation  policy  will  make  great  strides  in  validating  the 
effectiveness  of  the  Coast  Guard's  training  system.  There  is  no  doubt  that  the 
combination  of  a  strong  internal  evaluation  system  with  an  efficient  and  well-managed 
external  evaluation  system  will  ensure  that  Coast  Guard  training  effectively  provides 
essential  knowledge  and  skills  needed  to  meet  the  Service's  challenging,  multi-mission 
demands. 

Post-Training  Survey 
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Section  1:  Task  List 

Do  you  perform  the  following? 

YES 

m 

If  yes,  rate  your  confidence: 

high - 1 

Performance  Indicator  A 

■ 

H 

u 

H 

5 

■ 

Performance  Indicator  B. . .etc. 

■ 

M 

■ 

■ 

Section  2:  Training  Benefit 


1  -  Training  provided  complete  knowledge/skills  to  proficiently  perform  the 
task. 

2  -  Training  provided  a  strong  base  to  proficiently  perform  the  task. 

3  -  Training  was  of  little  or  no  benefit  to  proficiently  perform  the  task. 

4  -  Task  performed  at  this  unit,  but  my  duties  do  not  require  me  to  complete 
this  task. 

5  -  Task  not  performed  at  this  unit. 


Which  of  the  statements  above  is  most  true  of  the  following  tasks? 


Performance  Indicator  A 

■ 

m 

ifl 

H 

5 

Kfl 

Performance  Indicator  B. . .etc. 

m 

H 

Kfl 

Ifl 

Section  3:  Comments 

1 .  Are  there  any  tasks  performed  at  your  unit  that  you  feel  should  have  been 
addressed  in  this  course? 

2.  Is  there  anything  that  prevents  you  from  becoming  proficient  in  the  tasks  associated 
with  this  course  (e.g.,  tasks  are  automated,  do  not  perform  the  tasks  at  your  level  of  rank, 
outdated  procedures  or  policies  taught  in  the  course)? 

Figure  2:  Sample  of  Level-3  Survey  Instrument 
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